Neural networks trained with gradient descent often learn solutions of increasing complexity: the model first captures simple structure, then progressively incorporates finer details [AJB+17, KKN+19, ZSL25]. This incremental learning phenomenon, often visible as plateaus in the training loss separated by rapid transitions, has attracted significant theoretical attention. The most detailed analyses of incremental learning have been carried out for diagonal linear networks, including precise descriptions of transition times and plateau levels [Ber23, PF23]. This level of detail is possible because the training dynamics of these networks reduce to a mirror flow [WGL+20]. Mirror flows themselves feature incremental learning when initialized near the boundary of the domain of the mirror potential. This paper gives a rigorous description of this phenomenon for a broad class of mirror flows, thereby generalizing the previous analyses of diagonal linear networks.

The most forceful driver of advancements in the field of Machine Learning over the past decades has been the success of deep neural networks.

The most forceful driver of advancements in the field of Machine Learning over the past decades has been the success of deep neural networks.

As part of the effort to understand implicit bias of gradient descent in over-parametrized models, several results have shown how the training trajectory on the overparametrized model can be understood as mirror descent on a different objective. The main result here is a characterization of this phenomenon under a notion termed commuting parametrization, which encompasses all the previous results in this setting.

As part of the effort to understand implicit bias of gradient descent in over-parametrized models, several results have shown how the training trajectory on the overparametrized model can be understood as mirror descent on a different objective. The main result here is a characterization of this phenomenon under a notion termed commuting parametrization, which encompasses all the previous results in this setting.

We examine the continuous-time counterpart of mirror descent, namely mirror flow, on classification problems which are linearly separable. Such problems are minimised'at infinity' and have many possible solutions; we study which solution is preferred by the algorithm depending on the mirror potential. For exponential tailed losses and under mild assumptions on the potential, we show that the iterates converge in direction towards a \phi_\infty -maximum margin classifier. The function \phi_\infty is the horizon function of the mirror potential and characterises its shape'at infinity'. When the potential is separable, a simple formula allows to compute this function.

In recent years, the application of deep networks has revolutionized the field of machine learning, particularly in tasks involving complex data such as images and natural language. These models, typically trained using stochastic gradient descent, have demonstrated remarkable performance on various benchmarks, raising questions about the underlying mechanisms that contribute to their success. Despite their practical efficacy, the theoretical understanding of these models remains relatively limited, creating a pressing need for deeper insights into their generalization abilities. The classical theory shows that the latter is a consequence of regularization, which is the way to impose a priori knowledge into the model and to favour "simple" solutions. While usually regularization is achieved either by choosing simple models or explicitly adding a penalty term to the empirical risk during training, this is not the case for deep neural networks, which are trained simply by minimizing the empirical risk. A new perspective has then emerged in the recent literature, which relates regularization directly to the optimization procedure (gradient based methods). The main idea is to show that the training dynamics themselves exhibit self regularizing properties, by inducing an implicit regularization (bias) which prefers generalizing solutions (see [Vardi, 2023] for an extended review of the importance of implicit bias in machine learning). In this context, a common approach is to study simplified models that approximate the networks used in practice. Analyzing the implicit bias of optimization algorithms for such networks is facilitated but still might give some insights on the good performance of neural networks in various scenarios.

We examine the continuous-time counterpart of mirror descent, namely mirror flow, on classification problems which are linearly separable. Such problems are minimised `at infinity' and have many possible solutions; we study which solution is preferred by the algorithm depending on the mirror potential. For exponential tailed losses and under mild assumptions on the potential, we show that the iterates converge in direction towards a $\phi_\infty$-maximum margin classifier. The function $\phi_\infty$ is the $\textit{horizon function}$ of the mirror potential and characterises its shape `at infinity'. When the potential is separable, a simple formula allows to compute this function. We analyse several examples of potentials and provide numerical experiments highlighting our results.